An Evaluation of Multi-Modal User Interface Elements for Tablet-Based Robot Teleoperation

For robot teleoperation systems, tablet and smart phone user interfaces provide portability and accessibility to allow anyone anywhere to connect to the system quickly and easily. This can be highly advantageous for disaster-relief robot systems where timing is critical. However, the small screen size and unconventional input methods mean that traditional teleoperation user interface elements, such as multiple visual windows, keyboards and mice, are no longer effective. In this paper we propose and investigate multi-modal user interface design principles as a solution for effective tablet-based teleoperation. We present our tablet user interface that features multiple complementary modalities of control and feedback mechanisms. It allows the operator to perceive the robot’s environment and state by exciting their senses of vision, hearing and touch with overlaying visual displays, virtual 3-D audio and vibration feedback. The operator commands actions to the robot through the use of intuitive input methods utilising the touch screen and accelerometer. We analyse these design principles by performing a user study that focuses on finding what aspects of the multi-modal user interface affect the system performance during a navigation task. The study highlights advantages of multimodal user interfaces for robot teleoperation and makes several findings to be considered when developing similar systems.